This section describes a sample configuration for deploying CCC in a high availability configuration. CCC provides an open and flexible architecture that supports multiple configuration options, including high availability. The figure below illustrates a CCC configuration that provides all of the features inherent in a high-availability system.

If you want to use HA configuration, you need to use an external database.

By adding additional and standby components to the basic configuration you can create a high availability system that offers the following features:

Server redundancy

A true high-availability system needs to continue to operate, without interruption, in the event that one of the system components fails. To deliver this capability, it is recommended that you set up two worker nodes running CCC servers, one Master node for Kubernetes, two PostgreSQL database servers, and two NFS servers.

Load balancing

The Kubernetes master node takes care of load balancing and forwards requests to all CCC servers. It takes each request in order and routes it to one of the worker nodes running CCC servers. If one of the CCC servers goes down, the request is forwarded to the next server, with no impact to the end user. Kubernetes is configured to support persistent (or sticky) sessions, such that all requests for a specific user session are directed to the same CCC server. If a failover occurs, the user will be prompted to log in to another active server, and all future requests will be directed to the new server for the duration of the session.

Data replication

It is recommend that you configure PostgreSQL database to provide streaming replication of the database between an active and standby database server. Any HA implementation should include this, or a similar feature, to ensure uninterrupted access to the data stored in the CCC database in the event of a failure. It is recommended that you configure NFS server to provide streaming replication of CCC data between the active and standby NFS servers. Any HA implementation should include this, or a similar feature, to ensure uninterrupted access to the data stored on the NFS server in the event of a failure.

Failover protection

Kubernetes takes care of redirecting the users to an active CCC server in case one of the CCC servers on worker node goes down.

Database failover is managed by installing an application, such as keepalived, on each database server that allows you to use a virtual IP address to identify the active database server so that if the active server goes down, the standby server takes the IP address and becomes the active server. Notification of a failover is also required so that the databases can be re-synchronized when the failed database server is brought back online.

NFS server failover is managed by installing an application, such as keepalived, on each NFS server that allows you to use a virtual IP address to identify the active NFS server so that if the active server goes down, the standby server takes the IP address and becomes the active server.

Operations that are in progress when a failover occurs may fail.

Setting up File Sharing

To set up file sharing among CCC servers, follow these steps on each NFS server:

1. Run the enableNFSSharing.sh script on the NFS server that you've selected. This script will be available at the following path on a CCC server: CCC_packages/kubernetes/enableNFSSharing.sh. Following is the general command syntax for executing the enableNFSSharing.sh script:
   ./enableNFSSharing <NFSOption> <IPAddress(s)>

   Values allowed for are: 1 for NFS Server and 2 for NFS Client.

   If you execute the enableNFSSharing.sh script without any arguments or wrong arguments, you will get the following message:
   Usage: enableNFSSharing NFSOption[1: For NFS Server 2: For NFS Client] IPAddress.

   Enter valid IP addresses to avoid getting an error.

2. Run the following command on the NFS server by navigating to the folder where enableNFSSharing.sh script is copied: ./enableNFSSharing.sh 1 Server.List
   Here, Server.List is the list of all CCC servers to be setup in HA mode as NFS client, separated by a space.
   Example: ./enableNFSSharing.sh 1 20.10.10.10 30.10.10.10

3. Run the following script on the CCC servers to set up the NFS client:
   ./enableNFSSharing.sh 2 IP
   Here, IP is the IP address of NFS Server or Virtual IP address of the NFS server cluster in case CCC is set up in high availability.
   Example: ./enableNFSSharing.sh 2 10.10.10.10

To Unmount NFS Dir during CCC Uninstall

To unmount NFS directory during uninstall, CCC administrator runs following command as a root user:

umount -f -l /home/ccc/packages

This step should be performed only if CCC is set up in HA mode. This step should be performed on all CCC servers that have been set up as NFS client.

Tested Configuration

This section describes how to set up the HA configuration which was validated by the engineering team.

Master Node

The Kubernetes master node acts as the control panel for the CCC cluster running on Kubernetes. It is responsible for maintaining requests, keeping a count of desired replicas, load balancing, failover protection, persistent volumes, and network configurations of the cluster.

Worker Nodes (Redundant CCC application servers)

The CCC application is installed on two separate worker nodes that receive requests from the Master node. Requests are sent to the individual CCC servers from the master node. If one of the servers goes down, the request is forwarded to the other available server.

PostgreSQL servers

PostgreSQL is installed on two separate Linux workstations and is configured to use streaming replication. Keepalived is installed on each database server to provide failover and notification.

Server OS Installation and Network Configuration

An HA deployment requires seven separate servers which exchange data with each other, over the network, to operate as a unified system. When you configure CCC in HA, you are required to specify the network address (IP address or host name) of specific servers in the deployment. To simplify the deployment and avoid potential misconfigurations, the first step in deploying an HA configuration is to perform the following tasks:

• Install the operating system on each server used in the deployment.

• Configure the IP address or host name that will be used to identify the server.

• Assign a role to each server.

You must reserve two additional IP addresses - one for the database cluster and the other one for the NFS cluster. These IP addresses need to be specified during database configuration.

This document uses variables to identify each of the servers in the deployment, as listed in the following table After you complete the OS installation and network configuration, record the IP address or host name of each server on the following table, so that you can easily refer to this information when performing the various configuration tasks.

Server	Alias	IP Address/Host Name
Primary (active) PostgreSQL server	db_primary_IP_or_hostname
Standby PostgreSQL server	db_standby_IP_or_hostname
Keepalived database cluster virtual IP address	keepalived_virtual_IP
Primary (active) NFS server	nfs_primary_IP_or_host
Standby NFS server	nfs_standby_IP_or_hostname
Keepalived nfs cluster virtual IP address	keepalived_virtual_IP
CCC server 1 (Worker node 1)	ccc1_IP_or_hostname
CCC server 2 (Worker node 2)	ccc2_IP_or_hostname
Master node	Master_IP_or_hostname

Perform the following procedure on each server that will be used in the HA deployment:

1. Install the CentOS 7 distribution for your server architecture:

   a. CentOS 7 is available for download from http://wiki.centos.org/Download. It is recommended that you install only the base software (the Minimal installation option) to avoid installing unnecessary software that could present a security risk. Refer to the CentOS documentation for detailed installation instructions.

   b. After the installation is complete, reboot the system by entering the command systemctl reboot.

   c. Log in as root and enter the yum update command to install the latest updates.

2. Configure a static IP address or host name on the server and record the information in the table above. Refer to the CentOS documentation for detailed network configuration procedures.

Configuring and Setting up PostgreSQL Server in HA Mode

This section describes how to set up a PostgreSQL high availability (HA) cluster configuration consisting of a primary PostgreSQL server and a standby PostgreSQL server. The cluster is configured to use streaming replication. This procedure assumes PostgreSQL 14. Refer to the PostgreSQL documentation at https://www.postgresql.org/docs/14/index.html for more information.

Installing PostgreSQL

You require two standalone Linux servers: one for the primary, and one for the standby. The tested and documented configuration uses CentOS 7 and PostgreSQL 14. Other operating systems may work, although they are not tested, and may use different paths for some components.

Configuring the Primary PostgreSQL Database Server

To configure the primary PostgreSQL database server:

1. Log in as root to the server you identified as the primary PostgreSQL server and set the permissions for the session:
   su root
   umask 0022

2. Edit the /var/lib/pgsql/14/data/postgresql.conf file to uncomment and update the following entries, which are used to configure PostgreSQL to use streaming replication:

   listen_addresses = db_primary_IP_or_hostname>,<keepalived_virtual_IP
ssl = on
wal_level = hot_standby
checkpoint_segments = 32
archive_mode = on
archive_command = 'cp %p /tmp/%f'
max_wal_senders = 3
wal_keep_segments = 32
   

   listen_addresses is a comma separated list of the hosts the server will respond to. It must also include the keepalived virtual IP address for the database cluster (configured later), and may include other servers, if required. Access to the database is controlled by the pga_ hba.conf file, as explained in the next step.

3. Open the /var/lib/pgsql/14/data/pg_hba.conf file in a text editor and add an entry for CCC to the #IPv4 local connections section of the file. Add the following lines as the first entry in the section to allow connections from the CCC host:

   To ensure that CCC can successfully connect to the database, the entry must be the first lines in #IPv4 local connections section.

   host replication replicator <db_secondary_IP_or_hostname>/32 md5
   

   hostssl lunadirectordb lunadirector <ccc1_IP_or_hostname>/32 md5
   

   hostssl lunadirectordb lunadirector <ccc2_IP_or_hostname>/32 md5
   

   If both of your CCC servers reside in the same subnet, you can add a single line to the file to allow access from all devices in that subnet. Add an entry for hostssl lunadirectordb lunadirector /24 md5 as the last line in the var/lib/pgsql/14/data/pg_ hba.conf file.

4. Save and close the file.

5. Restart the PostgreSQL Service.

   systemctl restart postgresql-14.service

6. The connection between the CCC application servers and the PostgreSQL servers uses SSL, which requires that you create a server certificate. Enter the following commands to create a self-signed certificate with the correct owner and permissions. After entering the openssl req -new -text -out server.req -nodes command, you will be presented with a series of prompts asking you to specify the certificate attributes. The only important attribute is the server Common Name (CN), which must be set to the virtual IP of the database cluster. You must specify the virtual IP, since the same certificate will be used on the standby database server to handle failover.
   

   cd /var/lib/pgsql/14/data
openssl req -new -text -out server.req -nodes
openssl rsa -in privkey.pem -out server.key
rm -f privkey.pem
openssl req -x509 -in server.req -text -key server.key -out server.crt
chmod og-rwx server.key
chown postgres:postgres server.key
systemctl restart postgresql-14
   

7. Enter the following commands to set up the replication and CCC users and tables in the database:

   su - postgres -c "psql postgres postgres -c \"CREATE USER replicator REPLICATION LOGIN PASSWORD 'dbpass';\""
su - postgres -c "psql postgres postgres -c \"CREATE USER lunadirector encrypted PASSWORD 'password';\"", where password is the password the lunadirector user will use to access the database.

   Remember the password you specified for the lunadirector user. You will need it later when you configure CCC. This is the password that the CCC application server uses to connect to the PostgreSQL database cluster.

   su - postgres -c "psql postgres postgres -c \"CREATE DATABASE lunadirectordb OWNER lunadirector;\""

8. Enter the following command to configure the firewall (iptables) to allow the CCC servers to access the database. By default, PostgreSQL listens on port 5432 for connections:

   iptables -I INPUT 2 -p tcp -m tcp --dport 5432 -j ACCEPT

Configuring Standby PostgreSQL Database Server

To configure the standby PostgreSQL database server:

1. Log in as root to the server you identified as the primary PostgreSQL server and set the permissions for the session:
   su root
umask 0022

2. Edit the /var/lib/pgsql/14/data/postgresql.conf file to uncomment and update the following entries, which are used to configure PostgreSQL to use streaming replication:

   listen_addresses = '<db_standby_IP_or_hostname>,<keepalived_virtual_IP>'
ssl = on
wal_level = hot_standby
checkpoint_segments = 32
max_wal_senders = 3
wal_keep_segments = 32
hot_standby = on

3. Open the /var/lib/pgsql/14/data/pg_hba.conf file in a text editor and add an entry for CCC to the #IPv4 local connections section of the file. Add the following lines as the first entry in the section to allow connections from the CCC host:

   host replication replicator <db_secondary_IP_or_hostname>/32 md5 hostssl lunadirectordb lunadirector <ccc1_IP_or_hostname>/32 md5
hostssl lunadirectordb lunadirector <ccc2_IP_or_hostname>/32 md5

   To ensure that CCC can successfully connect to the database, the entry must be the first lines in #IPv4 local connections section.

   If both of your CCC servers reside in the same subnet, you can add a single line to the file to allow access from all devices in that subnet. Add an entry for hostssl lunadirectordb lunadirector <subnet_IP>/24 md5 as the last line in the var/lib/pgsql/14/data/pg_ hba.conf file.

4. Save and close the file.

5. Restart the PostgreSQL Service.

   systemctl restart postgresql-14.service

6. Copy the database from the primary PostgreSQL server to the standby PostgreSQL server. Copying the database deletes the /var/lib/pgsql/14/data directory that contains the postgresql.conf configuration file, so the following step includes backing up the directory and restoring it after copying the database.

   a. Ensure that you are not in the /var/lib/pgsql/14/data directory:

   cd

   b. Backup the PostgreSQL configuration files:

   cp /var/lib/pgsql/14/data/postgresql.conf /tmp/postgresql.conf.bak
cp /var/lib/pgsql/14/data/pg_hba.conf /tmp/pg_hba.conf.bak

   c. Stop the PostgreSQL service:

   systemctl stop postgresql-14

   d. Delete the /var/lib/pgsql/14/data/ directory. It will be restored later.

   sudo -u postgres rm -rf /var lib/pgsql/14/data/

   e. Replicate the database from the primary database server to the standby database server:

   sudo -u postgres pg_basebackup -h <db_primary_IP_or_hostname> -D /var/lib/pgsql/14/data -U replicator -v -P

   Where <db_primary_IP_or_hostname> is the IP address of host name of the primary database server. You are prompted to enter the password for the 'replicator' user. The password is 'dbpass'. The following message may be displayed. It can be ignored.

   Could not change directory to "/root": Permission denied.

   f. Enter the following commands to configure the recovery.conf file:

   sudo -u postgres bash -c "cat > /var/lib/pgsql/14/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=<db_primary_IP_or_hostname> port=5432 user=replicator
password=dbpass'
trigger_file = '/tmp/postgresql.trigger'
_EOF1_ "

   g. Restore the postgresql.conf and pg_hba.conf files from the backups:

   cp -f /tmp/postgresql.conf.bak /var/lib/pgsql/14/data/postgresql.conf
cp -f /tmp/pg_hba.conf.bak /var/lib/pgsql/14/data/pg_hba.conf
   

   h. Start the PostgreSQL service:

   systemctl start postgresql-14

7. Enter the following command to configure the firewall (iptables) to allow the CCC servers to access the database. By default, PostgreSQL listens on port 5432 for connections:

   iptables -IINPUT 2 -p tcp -m tcp --dport 5432 -j ACCEPT

Testing the PostgreSQL Database Cluster

To verify that streaming replication is configured correctly, you can create a table on the primary database and verify that it is replicated on the standby database. To test the PostgreSQL database cluster:

1. Create a table (named test) on the primary database:

   su - postgres -c "psql postgres postgres -c \"CREATE TABLE test (name char(10));\""

2. Verify that the table was replicated on the standby database:

   a. Login to the standby database server as root:

   su root

   b. Start PostgreSQL.

   systemctl start postgresql-14

   c. Connect to the database:

   su - postgres psql -d postgres

   d. List the tables in the database:

   \dt *.*

   If streaming replication is configured correctly, the test database table is listed in the output. If it is not, check your configuration and try again.

3. Delete the table (named test) on the primary database:

   su - postgres -c "psql postgres postgres -c \"DROP TABLE test;\""

4. Attempt to create a table (named test) on the standby database:

   su - postgres -c "psql postgres postgres -c \"CREATE TABLE test (name char(10));\""

5. Verify that the command fails with the following error:

   “ERROR: cannot execute CREATE TABLE in a read-only transaction”

Setting Up Keepalived On the PostgreSQL Servers

You must install the keepalived software on each PostgreSQL server to manage failover to the standby server in the event of an outage on the primary server. Once keepalived is installed and configured, if the primary server goes down, the standby server takes over as the new primary server, and the old primary server becomes the standby server. Keepalived allows you to configure a virtual IP address for the database cluster, so that database failover is transparent to CCC. For more information, refer to the keepalived documentation, available at keepalived.org.

To install and configure keepalived on the PostgreSQL database servers:

1. Install keepalived on both of the PostgreSQL database servers:

   yum install keepalived

2. Edit the /etc/keepalived/keepalived.conf file on the primary server as follows:

   ! Configuration File for keepalived
global_defs {
notification_email {
}
notification_email_from <email_address>
smtp_server <smtp_server_IP_or_hostname>
smtp_connect_timeout 30
router_id CCC_DB_MONITOR
}
vrrp_instance VI_1 {
state MASTER
interface <eth0 | eth1 | eth2 | ...>
virtual_router_id 51
priority 101
advert_int 1
authentication {
auth_type PASS
auth_pass PASSWORD
}
virtual_ipaddress {
<db_cluster_virtual_IP> dev <eth0 | eth1 | eth2 | ...>
}
}
   

   Where:

   Parameter	Description
   email_address	The email address used to send a notification message in the event of a failover.
   smtp_server_IP_or_hostname	The IP address or hostname of the SMTP server used to send the notification message.
   db_cluster_virtual_IP	The virtual IP address for the database cluster.
   eth0	eth1

   Replace all existing content so that your file contains only the following entries.

3. Edit the /etc/keepalived/keepalived.conf file on the standby server as follows, where:

   ! Configuration File for keepalived
global_defs {
notification_email {
}
notification_email_from <email_address>
smtp_server <smtp_server_IP_or_hostname>
smtp_connect_timeout 30
router_id CCC_DB_MONITOR
}
vrrp_instance VI_1 {
state BACKUP
interface <eth0 | eth1 | eth2 | ...>
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass PASSWORD
}
virtual_ipaddress {
<db_cluster_virtual_IP> dev <eth0 | eth1 | eth2 | ...>
}
notify_master /root/<path to notify_master script>
notify_backup /root/<path to notify_standby script>
}
   

   Where:

   Parameter	Description
   email_address	Specifies the email address used to send a notification message in the event of a failover.
   smtp_server_IP_or_hostname	Specifies the IP address or hostname of the SMTP server used to send the notification message.
   db_cluster_virtual_IP	Specifies the virtual IP address for the database cluster.
   eth0	eth1
   path to notify_master script	Specifies the path to the notify master server script. Deactivates the PostgreSQL service on the primary database server in event of failover.
   path to notify_backup script	Specifies the path to the notify backup server script. Alerts the secondary database that the primary has failed and switches the server from readonly to active.

   Replace all existing content so that your file contains only the following entries.

4. Enter the following command on both of the PostgreSQL database servers to configure the firewall (iptables) to allow multicast:

   iptables -I INPUT -i eth2 -d 224.0.0.0/8 -j ACCEPT

5. Start keepalived on both of the PostgreSQL database servers, beginning with the primary:

   systemctl start keepalived

6. Restart postgresql on both of the PostgreSQL database servers, beginning with the primary:

   systemctl restart postgresql-14

Testing keepalived

You can verify that keepalived is working by performing the following tasks on both database servers:

• View the logs in /var/log/messages.

• Run the following command to see if the virtual IP is bound where you expect it to be. In normal operation, the virtual IP is bound to the primary database server only:

  ip addr show <eth0 | eth1 | eth2 | ...>

CCC Application Server Setup and Configuration

Setting up and configuring the CCC application servers involves performing the following tasks on each server used to host the CCC application. To set up and configure the CCC servers:

1. Install CCC using Kubernetes.

   Provide the IP address for your postgresql HA setup in the configuration file (config-map.yaml) during CCC installation.

2. Run the following command to replicate the CCC application.

   
   kubectl scale --replicas=2 deployment ccc-deployment
   

   This command runs two CCC instances, one on each of the worker nodes.

3. Follow CCC container logs by running the following command on master node:

   
   kubectl logs –f 
   

4. Once the installation gets completed on both the pods, you can log in to CCC using one of the following options:

   https://master_node_ip
   https://master_hostname:30036